Vietnamese Text Retrieval: Test Collection and First Experimentations

In this paper we present the Vietnamese specialities in word boundary, morphology, part of speech that must be addressed in information retrieval relative tasks. Our experiments have shown how different types of Vietnamese index terms: " tiӃng " , words, compound words, combination of word and compound word contribute to Vietnamese text processing and retrieval. We also introduce our Vietnamese test collection on which experimentations have been done and report the method used to construct this test collection. 1. Vietnamese specialities Vietnamese is a monosyllabic language which uses a Latin alphabet with accents on the vowels to create new tonalities such " ă " , " â " , " ê " , " ô " , " ѭ ". Vietnamese have six different tons which modify the meaning of the words, for example: ma (phantom), má (cheek), mà (but), mҧ (tomb), mã (code), mҥ (rice seedling). Therefore, we can not use ASCII to encode Vietnamese characters. Instead, there are many character-sets have been using in Vietnamese electronic text such as: ABC, TCVN, VNI, UTF-8…and UFT-8 is the most common nowadays. Consequently, we may need a normalization of encoding prior to the phase of indexing. Vietnamese has a special linguistic unit called " tiӃng " (equivalent to hanzi of Chinese) which is similar to traditional morphemes in respect of content and similar to traditional syllables in respect of form [7]. A Vietnamese word consists of one or more " tiӃng " separated by space, for example: " sách " (book), " dӳ liӋu " (data), " xă hӝi chӫ nghƭa " (socialist) etc. Therefore, the whitespaces can not be used to identify the word boundary. This is a challenge for both Vietnamese Natural Language Processing (NLP) in general and Vietnamese text retrieval in particular. We will discus in details how different kinds of Vietnamese index terms contribute to the precision and recall of IR system in the experimentation section. Vietnamese word is morphologic invariant: The word form is unchanged to its different grammatical roles in the sentence like that in Euro-Indian languages. Therefore, the lemmatization in index phase is not necessary for Vietnamese words. However, there are some exceptions in the processing of which morphologic normalization is needed. These exceptions are raised by two cases: the first is, the usage of vowels i and y is interchangeable in some circumstances such as " bác sƭ " and " bác sӻ …